{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Notebook for Classification Metrics\n",
    "\n",
    "**Author:** Matthias Quinn\n",
    "\n",
    "**Date Began:** October 23rd, 2022\n",
    "\n",
    "**Goals:** To learn more about Thieu's classification metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Import the necessary libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from permetrics.classification import ClassificationMetric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Demonstration of Classification Class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "y_true = [0, 1, 0, 0, 1, 0]\n",
    "y_pred = [0, 1, 0, 0, 0, 1]\n",
    "\n",
    "evaluator = ClassificationMetric(y_true, y_pred)\n",
    "cm, imap, imap_count = evaluator.confusion_matrix()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2 0 0]\n",
      " [1 1 1]\n",
      " [1 0 2]]\n",
      "{'ant': 0, 'bird': 1, 'cat': 2}\n",
      "{'ant': 2, 'bird': 3, 'cat': 3}\n"
     ]
    }
   ],
   "source": [
    "y_true = [\"cat\", \"ant\", \"cat\", \"cat\", \"ant\", \"bird\", \"bird\", \"bird\"]\n",
    "y_pred = [\"ant\", \"ant\", \"cat\", \"cat\", \"ant\", \"cat\", \"bird\", \"ant\"]\n",
    "\n",
    "evaluator = ClassificationMetric(y_true, y_pred)\n",
    "cm, imap, imap_count = evaluator.confusion_matrix()\n",
    "print(cm)\n",
    "print(imap)\n",
    "print(imap_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Looks simple enough."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Multiple Metrics:\n",
    "\n",
    "It looks like there are 3 ways to request multiple metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "1. Retrieve a list of metrics by using a loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'PS': 0.72222}, {'RS': 0.66667}, {'LS': 2.14815}, {'SS': 0.82222}]\n"
     ]
    }
   ],
   "source": [
    "## 1. Get list metrics by using loop\n",
    "list_metrics = [\"PS\", \"RS\", \"LS\", \"SS\"]\n",
    "\n",
    "list_results = []\n",
    "for metric in list_metrics:\n",
    "    list_results.append( evaluator.get_metric_by_name(metric) )\n",
    "print(list_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "2. By using the created class and a list of metrics you'd like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'PS': 0.72222, 'RS': 0.66667, 'LS': 2.14815, 'SS': 0.82222}\n"
     ]
    }
   ],
   "source": [
    "dict_result_2 = evaluator.get_metrics_by_list_names(list_metrics)\n",
    "print(dict_result_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "3. By using a dictionary of metrics with their associated parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'PS': 0.625, 'RS': 0.66667, 'LS': 2.14815, 'SS': 0.84167}\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# * 3. Get list metrics by using function and parameters\n",
    "dict_metrics = {\n",
    "    \"PS\": {\"average\": \"micro\"},\n",
    "    \"RS\": {\"average\": \"macro\"},\n",
    "    \"LS\": None,\n",
    "    \"SS\": {\"average\": \"weighted\"},\n",
    "}\n",
    "dict_result_3 = evaluator.get_metrics_by_dict(dict_metrics)\n",
    "print(dict_result_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Functional Style Calling:\n",
    "\n",
    "This is a traditional way to call a specific metric you want to use. Everytime you want to use a function, you need to pass `y_true` and `y_pred`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision: 0.72222, 0.72222, 0.72222\n"
     ]
    }
   ],
   "source": [
    "evaluator = ClassificationMetric()\n",
    "\n",
    "# * 3.1 Call specific function inside object\n",
    "# * , each function has 2 names like below\n",
    "\n",
    "ps1 = evaluator.precision_score(y_true, y_pred)\n",
    "ps2 = evaluator.PS(y_true, y_pred)\n",
    "ps3 = evaluator.PS(y_true, y_pred)\n",
    "print(f\"Precision: {ps1}, {ps2}, {ps3}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.6 ('MLDL')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "f294934ec600687241badbba13b5ef9e354863a1e6583340aa87f207e4656842"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}